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ABSTRACT 

This article describes the results of a follow-up investigation to a study in which researchers proposed a model for 
implementing a college-wide initiative in which ten essential learning outcomes students acquire from curricular 
and extra-curricular learning experiences were developed and implemented at the university. Using descriptive and 
correlational statistics, and an analysis of qualitative data researchers identified pedagogicalfactors that supported 
student competence with essential learning outcomes. This follow-up study investigated the role of the syllabus as a tool 
for promoting ELO competence, and the results ofthis study also serve asfurther evidence ofthe pedagogical practices 
that support ELO competence in students. 

Collectively, the data from this second study reveal that intentional practices on the part of faculty and professional 
staff, enhanced student ELO competence (Cydis, Galantino, Hood, Padden, and Richard, 2015). When researchers 
compared the results of the pre and post student self-perception questionnaires, an increase of 0.25, (p < 0.05) was 
found in round one of data collection, and an increase of 0.34 (p < 0.05) in round two indicating an increase in stu¬ 
dents’ self-perception oftheir ELO competencefrom the beginning to the end ofthe semester. In the second iteration of 
the learning community and data collection, researchers further analyzed the content of participant syllabi using an 
adapted version of The Syllabus Rubric (Palmer, Bach and Streifer, 2014) andfound that while explicit and inten¬ 
tional efforts to effectively integrate EL Os did in fact promote student perceived EL 0 competence, a learner-centered 
syllabus alone, did not necessarily correlate with increases in student perceived competence in the essential learning 
outcomes targeted in respective courses. 


INTRODUCTION 

The discussion that follows focuses on 1) a model for ELO 
integration, 2) the role of syllabus design in supporting 
student learning, 3) student perceptions of competence 
with essential skills and 4) how syllabus design, the in¬ 
tegration of essential learning outcomes as defined by 


LEAP (2005) and AAC&U and a focus on high impact 
pedagogical practices support student achievement. The 
discussion will include consideration of formative assess¬ 
ment and closing the feedback loop to support student 
learning. Most importantly the role of the syllabus and its 
value as well as limitations to promoting competence in 


Journal of Learning in Higher Education 


81 



Susan Cvdis. MarvLou Galantino. Carra Leah Hood. Marv Padden. & Marc Richard 


students is considered. Thus, we explore the relationship 
between syllabus design and faculty and student percep¬ 
tion of ELO achievement. 

BACKGROUND 

Through the scholarship of teaching and learning faculty 
participate in a collaborative, meaningful, and energizing 
dimensions of professional development in higher educa¬ 
tion. Focusing on the scholarship of teaching and learn¬ 
ing is not only an opportunity to reflect on pedagogy and 
redesign opportunities for student learning, but also ad¬ 
vances institutional agendas in higher education. Through 
faculty development and the cycle of feedback and assess¬ 
ment, the scholarship of teaching and learning can be 
recognized and rewarded as an important contribution to 
the institution’s educational mission (Hutchings, Huber, 
and Ciccone, 2005). Faculty learning communities cre¬ 
ate opportunities to support program development and 
create more ambitious expectations for student learning 
(Hutchings, Huber, and Ciccone, 2005). Through this re¬ 
search project, faculty focused on the integration of essen¬ 
tial skills to promote student competence, inspired by a 
growing initiative entitled, Liberal Education and Amer¬ 
ica’s Promise (LEAP). The LEAP initiative, identified by 
the Association of American Colleges and Universities 
(AAC&U) has been influential in promoting academic 
development and student learning in higher education. 
The collaborative effort among colleges and universities to 
promote essential skills and learning outcomes students 
need for success in the 21 st century known as the LEAP 
initiative is an excellent illustration of opportunities for 
faculty collaboration and cause for engagement in the 
scholarship of teaching and learning. Through meaning¬ 
ful opportunities for scholarly work, faculty can focus on 
the integration of Essential Learning Outcomes (ELOs) 
and promote student competence with essential skills. 
The faculty learning community in this project aligns 
with the LEAP initiative includes a focus on the integra¬ 
tion of Essential Learning Outcomes. 

Essential Learning Outcomes 

The LEAP (2005) initiative provided research and re¬ 
sources for authentic assessment of the essential learning 
outcomes (ELOs) students acquire as a result of high- 
impact instructional practices and performance-based 
opportunities for students to demonstrate achievement 
of targeted outcomes. These practices include engaging 
students in meaningful, authentic learning that capital¬ 
izes on a constructivist approach and yields opportunities 
for students to gain greater metacognitive understandings 
about their own learning (McDermott, 1991). Through 
this approach, students develop competence through op¬ 


portunities to build capacities for life-long learning by 
applying learning to its intended purposes (Schneider, 
2013). As higher education’s purpose evolves from provid¬ 
ing general knowledge to building competencies (Lozano, 
Boni, Peris, Hueso, 2012), this approach aims to support 
student competence with essential skills. Recognizing the 
need to identify an effective model that includes key fea¬ 
tures for supporting ELO integration, researchers relied 
on a framework for faculty to integrate ELOs into their 
courses, developed in a previous research study. This mod¬ 
el for ELO integration focused on six themes to support 
student learning of essential skills identified in the discus¬ 
sion that follows (Authors, 2015). 

ELO Integration 

Model for ELO Integration. The cycle of teaching, 
learning, and assessment, as identified by the Middle 
States Commission on Higher Education (MSCHE), 
identifies a process for meaningful instruction. This pro¬ 
cess begins with clearly articulating learning outcomes, 
then providing purposeful opportunities for students to 
achieve those learning outcomes, and thirdly, assessing 
the effectiveness of the process to further enhance and 
drive instruction. Using an approach based on the process 
identified by MSCHE, the authors focused on a purpose¬ 
ful approach for integrating essential learning outcomes 
into the curriculum and proposed a model for ELO in¬ 
tegration. This model focuses on six themes: facilitation, 
integration, utility, connection, awareness, and reflection, 
identified and further described in Figure 1. 

The first of these themes emphasizes the role of the in¬ 
structor in creating opportunities for students to develop 
confidence with essential skills. This can include planning 
meaningful learning experiences and acting as an adviser 
while students engage in the experiential learning task. As 
they participate in this learning experience, it becomes an 
opportunity for students to develop an awareness of the 
utility of a particular essential learning outcome. Util¬ 
ity is identified as a second feature that supports ELO 
integration. As faculty members facilitate the integra¬ 
tion of ELOs and meaningful ways for students to learn, 
utility serves as an illustration of an intentional stance 
on the part of faculty in promoting student competence 
with ELO’s. Intention is identified as the third feature of 
ELO integration. Through this intentional integration of 
ELO’s, faculty can promote opportunities for reflection, 
which is identified as another feature of ELO integration. 
Engaging in the process of reflection on the part of the 
faculty member, as well as the student, promotes aware¬ 
ness of the ELO competence being developed. This aware¬ 
ness is identified as another feature of ELO integration. 
Finally, through integrating an ELO into a meaningful 
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Figure 1 

Features of ELO Integration 

J5ix Theme Fou nd to be Features of ELO Integration that Support Student Competence with EL Os 

(Cydis. Galaarico. Hood, Padden, Richard. 2015) 

Facilitation. This theme includes a reference to faculty members' comfort level or knowledge about ELO 
instruction and the role that teaching plays in supporting student competency with ELOs. This theme might also 
identify the need for faculty to guide or facilitate ELO acquisition in students through planned or incidental 
advising. 

Intention. Faculty participants identified a need or value for explicit emphasis of ELOs on the part of 
instructors to support student understanding. References to this theme include statements regarding repeated 
presentations of ELOs to students, revisiting the ELOs with students and or providing continuity in discussing 
the ELOs with students. 

Connection. This theme includes instances in which students connect one of the ELO to either another 
ELO or possibly connect one ELO intellectually or experienhally over time. For example, appreciating ELO 
knowledge from the beginning of the semester to the end of the semester. It might also include students 
recognizing continuity or a recursivorwt in one or more ELOs. Finally ibis theme included statements referring 
to linking concepts in a manner that demonstrates one or more ELOs. These concepts might also include course 
content or concepts learned in pragmatic experiences. 

Utility. In regard to this theme, faculty referred to the intended use of the ELO, a value for die authentic 
purpose of the ELO in the future. It might also refer to preparation of and or application of the ELO beyond the 
scope of the classroom This theme included references to the ELOs and Life-long learning. 

Reflection. According to FUviH Keogh, and Walker (1985, p. 19), "reflection m the context of learning is 
a generic term for those intellectual and affective activities in which individuals engage to explore their 
experiences in order to lead to new understandings and appreciations”. For this reason all references that coincide 
with this definition were identified within the theme of reflection. Additionally, references to this process of 
reflection and documentation of the behavior were also considered references to this theme. 

Atvareness. This theme emerged from statements that illustrated students' ability to identify one or more 
ELOs through a learning experience or a life experience, which heightened or contributed to recognition of ELO 
competence. This experience was interpreted as one that illuminated students' abilities as it related to ELO 
competence. 


learning experience, students have the opportunity to 
connect the ELO to other areas of their learning, either 
within their courses or to real-world experiences. The re¬ 
searchers note that the relationship among the themes is 
synergetic in nature and they co-exist interdependently, 
supporting effective ELO integration and student devel¬ 
opment of ELOs. The researchers report that the syner¬ 
getic relationship of these themes as shown in Figure 2, 
contributes to an effective model of ELO implementation 
and supports ELO competence on the part of the students 
(Author, et al, 2015). 

This proposed model integrated the use of Significant 
Learning Experiences (Fink, 2013) which focuses on the 


integration of foundational knowledge that is meaning¬ 
ful to students and supports their ability to and awareness 
of how to apply this knowledge to relevant real-world ex¬ 
periences. Similarly, to facilitate the intentional focus on 
ELO integration, researchers identified six themes that 
emerged from a focus on ELO integration as shown in 
Figure 2 (Author, et. al, 2015). The model offers a method 
for accomplishing objectives through meaningful learn¬ 
ing experiences and supports student competence through 
the application of skills and foundational knowledge. Stu¬ 
dents learn to apply the knowledge and skills they acquire 
in order to connect the learning to real world experiences. 
Students develop an understanding of key concepts, ap- 
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Figure 2 

Interdependence of ELO Themes 
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Formative assessment is defined as, 


“a process in which teachers and students engage 
in during instruction and which ongoing teach¬ 
ing and learning is adjusted through feedback for 
the purpose of improving student achievement 
and competence with instructional outcomes,” 
(Council of Chief State School Officers, 2014). 

The use of formative assessment techniques offers oppor¬ 
tunities to further support ELO integration. The authors 
argue that formative assessment is essential to successfully 
supporting student achievement of these 21st-century 
competencies and has a central role in creating a culture of 
learning (Council of Chief State School Officers, 2014). 
Techniques for formative assessment are identified by An¬ 
gelo and Cross (20014), as Classroom Assessment Tech¬ 
niques (CATs), in which faculty can administer a chosen 
technique and garner information to support students’ 
awareness, as well as feedback. The use of a formative as¬ 
sessment technique such as those identified by Angelo 
and Cross, offers opportunities for supporting ELO inte¬ 
gration. While the use of formative assessment techniques 
serves as an intentional strategy for ELO integration, so 
too does the articulation of ELOs explicitly stated as part 
of the design of the course syllabus. 

Syllabus articulation 

Researchers identify the syllabus as a tool for facilitating 
integration through clear, intentional, and explicit articu¬ 
lation of the essential skills and competencies students will 
acquire though meaningful learning experiences in a giv¬ 
en course. Instructors should use course syllabi not only to 
outline the content of the course and provide assignment 
descriptions, but also to articulate the general competen¬ 
cies students gain from engaging in the course (Parkes & 


Harris, 2002). Students report that syllabi should be com¬ 
prised of essential items including competencies acquired 
from the course (Appling, Gancar, Hughs & Saad, 2012). 
Parkes and Harris (2012) report that the syllabus can 
serve as a learning tool and a well-designed syllabus can 
facilitate student learning beyond the scope of the infor¬ 
mation that is intended for the course. Syllabi often omit 
important information related to what students will learn 
and the learning strategies students will secure as a result 
of the course (Singham, 2007). On the contrary, research¬ 
ers report that a syllabus can potentially help students de¬ 
velop professional skills and knowledge relevant to future 
careers (Parkes and Harris, 2012). 

Garavalua, Hummel, Wiley & Huitt found that faculty 
members and students prefer a more comprehensive syl¬ 
labus (1999). Students consider specific goals and objec¬ 
tives to be important factors, more so than faculty. As a 
result, faculty should be reflective about their syllabi and 
recognize the importance of fully explaining components 
necessary for successful performance in the course (Ga¬ 
ravalua, Hummel, Wiley & Huitt, 1999). This appears to 
support the value of reflection and intention as it relates 
to ELO integration and suggests further consideration for 
supporting student competence, as well as students’ per¬ 
ceptions of learning. Just as articulating relevant ELOs in 
the syllabus as an intentional focus on ELO integration, 
considering the self-perceptions students possess regard¬ 
ing ELO competence further serves as an opportunity for 
ELO integration. 

Student perceived competence 

Walker (2008) found that student reflection and percep¬ 
tion of learning promoted deeper understanding of in¬ 
tended student learning outcomes. Through opportuni¬ 
ties to reflect on learning and develop awareness, students 
demonstrate competence with learning outcomes. This 
serves as a prudent alternative to other assessment mea¬ 
sures that offer opportunities to gather knowledge about 
how students receive, interpret, and apply concepts, thus 
offering insight into multiple realms of learning (Walker, 
2008). For example, students’ perceptions of assessment 
have been shown to have an indirect effect on students’ 
competence and learning outcomes through students’ 
self-efficacy as a result of the reflective process. Walker 
(2008) found that students perceive the most important 
learning as the essentials not directly taught in the class¬ 
room, but those that align with activities or experiences 
outside the classroom. Study findings highlight which as¬ 
sessment characteristics, in particular meaningful, high 
quality feedback, positively influence students’ learning, 
and contribute to the effectiveness of competence-based 
education (Dinther, Dochy, Segers & Braeken, 2014). 
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Hence, the purpose of this study was to examine the syl¬ 
labus as a tool for supporting ELO integration and de¬ 
termine the impact on faculty and student perception of 
ELO achievement. The features of the syllabus for consid¬ 
eration included articulation of targeted goals including 
ELOs, a focus on significant learning experiences, assess¬ 
ment measures and the tone of the syllabus. 

THE RESEARCH STUDY (METHODOLOGY) 
Study Design 

A mixed methods approach included the use of multiple 
methods of analysis to gather and analyze various data 
sets relevant to this investigation. Using content analysis, 
a method aimed at examining features of a body of mate¬ 
rial, (Leedy and Ormrod, 2005), and a researcher created 
instrument adapted from The Syllabus Rubric (Palmer, 
Bach and Streifer, 2014), researchers analyzed the learner- 
centered features of faculty syllabi. Using thematic analy¬ 
sis, researchers explored the presence of themes identified 
in a previous study, known as “phenomena”, present in 
the experiences and explanations of faculty who designed 
syllabi to facilitate course competencies. (Authors, 2015). 
Using a descriptive analysis, researchers examined the pre 
and post student perceptions of ELO competence devel¬ 
oped by the university [Insert university hyperlink, Appen¬ 
dix A). 

Setting/Participants 

The setting of the study included a mid-sized state uni¬ 
versity with a student population of approximately 8,674. 
The university is a public, liberal arts university offer¬ 
ing undergraduate and graduate degrees in liberal arts 
and professional studies. The student population is 59% 
women and 41% men. Seventy-three percent of the stu¬ 
dent population are Caucasian, 10% Hispanic, 6% Afri¬ 
can American, and 5% Asian. Thirty-six percent of the 
undergraduate students are identified as low-income. The 
faculty includes 315 members, 45% male and 55% female, 
with 90% holding terminal degrees. Twenty-two percent 
hold professor status, 42% associate professor,34% as¬ 
sistant professor, while 2% are instructors. Faculty par¬ 
ticipants in this study were selected using criterion-based, 
non-probability sampling which “requires that one estab¬ 
lish the criteria, bases or standards necessary for units to 
be included in the investigation, one then finds a sample 
that matches these criteria” (Merriam, 1998, p. 48). Proce¬ 
dures for soliciting participants included a call for faculty 
via the university email system and required submission 
of a letter of interest for participation in the collaborative 
learning community. The goals of the learning commu¬ 


nity included collaboration among participants regarding 
strategies for supporting ELO competences in students. 
This included a focus on ELO integration into courses, as¬ 
signments, and co-curricular activities. More specifically, 
collaboration focused on redesigning participant syllabi 
to reflect ELO integration, creating significant learning 
experiences (Fink, 2013), the use of formative assessment, 
and a focus on the six themes that emerged and were iden¬ 
tified in the research study conducted during the first 
round of this learning community (Author, et ah, 2015). 
Facilitators reviewed letters of interest and selected facul¬ 
ty based on the goals of the project and the level to which 
they aligned with the goals identified in their respective 
letters of interest. Fourteen faculty members and profes¬ 
sional staff were selected as participants of this learning 
community which was referred to as the ELO Study 
Group - Round Two. As members of the Study Group, 
participants consented to this University Institutional 
Review Board (IRB) approved research study. 

Prior to data collection, faculty read and discussed rel¬ 
evant research and participated in a four-hour workshop 
relating to the pedagogical goals including meaningful 
learning experiences, formative assessment techniques, as 
well as syllabus and course design. Faculty administered 
student ELO self-perception surveys at the beginning and 
end of each semester. They also participated in collabora¬ 
tive meetings throughout the 2014-15 academic year. As 
the spring semester progressed, the study group met to 
discuss progress with supporting ELO competence in 
students and providing formative feedback. Additionally, 
faculty engaged in focus groups at the end of the semes¬ 
ter, completed surveys to articulate their experiences, and 
partook in individual interviews with the facilitators. In 
addition to student survey data and faculty focus groups, 
researchers collected faculty syllabi at the end of the se¬ 
mester and administered surveys to faculty to gather fac¬ 
ulty perspectives regarding their experiences. 

Data Collection Instruments: 

ELO Student Perception Survey 

Data was collected from students using the ELO Student 
Perception Survey Appendix B, where students rated 
themselves based on their perceived level of competence 
with each ELO. Using a 3-point Likert scale, student par¬ 
ticipants indicated whether they were (1) aware, (2) com¬ 
petent or (3) skilled with each respective ELO at baseline 
and the end of the semester. 
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Faculty Participant Syllabi 

Faculty members at the institution in this research setting 
are not required to utilize any specific format, however 
faculty participants were presented with various suggest¬ 
ed models for articulating ELO integration within the 
syllabus and several sample syllabi were provided to serve 
as examples of ELO integration. After the workshop and 
readings, faculty submitted an updated syllabus that in¬ 
cluded changes made to support student competence with 
ELOs. 

Faculty Interviews 

Researchers conducted post interviews with each faculty 
member. Interview questions (Appendix C) were de¬ 
signed to investigate various features of instructional ped¬ 
agogy aimed at creating a learner centered environment 
and promoting ELO competence in students. 

Faculty Survey 

Faculty completed an electronic post-survey (Appendix 
D) to investigate additional measures used to promote 
ELO competence in students that were not explicitly 
stated in either the faculty member’s syllabus or post in¬ 
terview. Six statements described various techniques that 
might have been used to promote ELO competence and 
faculty were asked to indicate any or all that were used to 
promote ELO competence. Each of the six statements re¬ 
lated to one of the six themes (Figure 1) identified as those 
that promoted ELO competence in the conceptual model. 
(Author, et ah, 2015). 

Data Analysis 

ELO Student Perception Survey 

Using Microsoft Excel (2016), a quantitative analysis of 
pre and post student perception survey scores determined 
mean differences for each ELO. A paired t-test identified 
the level of significance of the difference measured for 
each and was performed to determine the level of signifi¬ 
cance in ELO integration (Social Science Statistics, 2016). 

Faculty Syllabi 

Using the syllabus rubric shown in Appendix E, research¬ 
ers analyzed various features of the syllabi to identify 
low, moderate, or strong levels of a learner-centered ap¬ 
proach to instruction. These features included articula¬ 
tion of ELOs, Goals & Objectives, Assessment Activities, 
Significant Learning Experiences, and the Tone of the Syl¬ 


labus. The instrument used to analyze faculty syllabi was 
adapted from The Syllabus Rubric (Palmer, Bach & Strei- 
fer, 2014). The adaptations made to the Syllabus Rubric 
included adding an indicator for ELO integration and 
replacing the indicators that assessed the classroom envi¬ 
ronment with those indicators that assessed the tone of 
the syllabus. The purpose for these changes were to ensure 
better alignment with the goals of the ELO study group 
and the evaluation instrument used to assess the revised 
syllabi. The adapted rubric resulted in a criterion-based 
instrument that researchers used to evaluate syllabi. The 
process was a multi-step analysis in which each researcher 
initially scored five sample syllabi by rating the indicators 
as either (3) strong, (2) moderate, or (1) low/no evidence. 
Researchers then discussed individual scores as part of 
a norming process. Syllabi then received a total score by 
each researcher applying the same procedure used during 
the norming process. The scores were collated and the re¬ 
searchers discussed each indicator to ensure consistency 
among scores and inter-rater reliability of scores. An anal¬ 
ysis of the level of agreement among raters was conducted 
and a Cohen Kappa coefficient was determined to affirm 
inter-rater reliability and validity of scores. Overall, rubric 
scores indicated either a Content-Centered, Transitional, 
or Learner-Centered syllabus. A learner-centered syllabus 
(Bain, 2004) is defined as one that clearly communicates 
the learning students will gain and what is required to 
achieve that learning, along with a syllabus that scores 
between 35 and 45 is identified as a learner-centered syl¬ 
labus. A syllabus that scores between 25 and 34 is consid¬ 
ered transitional, while a score of 15 to 24 is identified as 
content-centered. 

Faculty Interviews Content Analysis Procedure 

Researchers conducted a content analysis process to ex¬ 
plore faculty interviews and identify the presence and fre¬ 
quency of the six themes identified by Author et al. (2015) 
as those that promote ELO competence in students. This 
process included first transcribing the interviews and then 
reading the interviews to glean a collective sense of overall 
content. In the third phase, researchers used HyperRE- 
SEARCH (2014) to code faculty statements that included 
references to one of the six themes. The researchers then 
reviewed the data to ensure a consensus and finally, the 
data was represented visually to reflect the frequency of 
faculty references to the themes found to promote ELO 
competence in students. 

Faculty Survey 

Responses to the faculty survey were tallied to identify the 
percentage of respondents used to promote ELO compe¬ 
tence in students not explicitly stated in either the faculty 
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member’s syllabus or the faculty post interview. Each of 
the six statements describes a technique that might have 
been used to promote ELO competence. The percentage 
was determined for each technique by comparing how 
many respondents identified the technique used to pro¬ 
mote ELO competence to the total number of thirteen 
respondents. The survey asked respondents to identify the 
frequency with which they intentionally addressed ELOs 
and also to identify any and all additional techniques used 
to promote ELO competence. The survey allowed respon¬ 
dents to select as many theme related statements that ap¬ 
plied to their experiences. Therefore, the researchers did 
not intend to develop a data set that was discrete in na¬ 
ture, but rather this data was intended to be continuous 
and inclusive, representing all strategies used by faculty 
participants to promote ELO competence in students. 

Content Analysis 

Researchers determined a Pearson product-moment cor¬ 
relation coefficient to assess the relationship between the 
integration techniques reported in the faculty survey and 
the frequency with which faculty referenced the themes 
identified as those that support student ELO competence. 


A second Pearson product-moment correlation coefficient 
was computed to assess the relationship between the syl¬ 
labus ratings determined to identify the learner-centered 
nature of the syllabi and the mean increases identified in 
the student survey of ELO competence. 

Results 

After researchers presented participants with IRB in¬ 
formed consent forms, data was collected from fourteen 
faculty participants and 323 students. In the ELO Stu¬ 
dent Perception Survey overall scores on the post-test 
were higher with a mean difference of 0.23 (p < .05) (Fig¬ 
ure 4). These results show an increase in student perceived 
ELO competence over the course of the semester. The dif¬ 
ferences in the means for the ELOs in each course showed 
increases ranging from 0.09 to 0.34 (Table 1). Further 
analysis revealed the mean difference of the ELOs target¬ 
ed by faculty for integration into their respective courses 
showed an increase of 0.34 (j> < .05). 
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Table 1 

Difference in Pre and Post Means for each ELO 


Creativity and Innovation (CS) 

Critical Thinking (CT) 

Program Competence (PC) 

Communication Skills (CS) 

Information Literacy (IL) 

Teamwork (TW) 

Global Awareness (GA) 

Adaptation & Change (AC) 

Quantitative Reasoning (QR) 

Ethical Reasoning (ER) 

Mean 

Diff 

0.30 

0.31 

0.23 

0.34 

0.17 

0.13 

0.27 

0.19 

0.30 

0.09 

t 

7.63 

7.30 

5.86 

8.53 

4.28 

3.21 

5.23 

4.51 

7.81 

2.45 

P 

<0.0001 

<0.0001 

<0.0001 

<0.0001 

<0.0001 

0.0015 

<0.0001 

<0.0001 

<0.0001 

0.0147 


Faculty Syllabi 

The content analysis of syllabi using the adapted syllabus 
rubric revealed a mean total score that ranged from 21.4 to 
43 on the sixteen existing indicators (Appendix E). Scores 
ranging from zero to twenty-five reflect a content-centered 
focused syllabus as determined by the instrument, while 
scores ranging between twenty-five and thirty-five indi¬ 
cates the transitional stage in terms of a content-centered 
versus a learning-centered syllabus. Moreover, scores be¬ 
tween thirty-five and forty-five reflect a learning-centered 
syllabus. Based on the means established for each syllabus, 
one of the faculty participant syllabi was identified as a 
content-centered syllabus, six were recognized as transi¬ 
tional and nine were classified as learning-centered syllabi 
(Table 2). 

Faculty Interviews 

The content analysis of faculty interviews revealed the 
presence of 101 instances of faculty references to one of 
the six themes identified as those that promote ETO com¬ 
petence in students. (Figure 1). All six themes were refer¬ 
enced by faculty during the interviews and the frequency 
of references to each of the six ranged from four to twenty- 
eight as shown in Table 3. The themes of Intention and 
Facilitation were most frequently referenced by faculty, 
making up 28% and 29% respectively, while Connection 
had the fewest references, equaling 4% of the total refer¬ 
ences. Other themes included Utility, encompassing 10% 
of the references, Awareness with 11% and Reflection with 
17% of the references made by faculty. 


Faculty Survey 

Faculty survey data revealed that all eleven respondents 
identified one or more additional measures to promote 
ELO integration during the course of the semester. All 


Table 2 

Learner-Centered Ratings of 
Course Syllabi 


Syllabus 

Mean 

Syllabus 

Rating 

|n=5) 

A 

21.4 

B 

27.8 

C 

30.2 

D 

30.6 

E 

31.2 

F 

34.4 

G 

35 

H 

36.4 

i 

36.6 

j 

38.4 

K 

38.8 

L 

40.2 

M 

40.2 

H 

42.6 

i 

43 

j 

43.4 



Content-Centered 

15-25 

Transitional 

25-35 

Learner-Centered 

35-45 
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Table 3 

Faculty References to Themes of ELO Integration and Features of ELO Integration 



Intention 

Intent 1-2x 

Intent 3-5x 

Intent >5x 

Connection 

Reflection 

Utility 

Facilitation 

Awareness 

Interview Coding 

28% 


4% 

17% 

10% 

2S c o 

11% 

n=101codes 

28 

4 

17 

10 

29 

11 

Faculty Survey 

100% 

27% 

46% 

27% 

18% 

56% 

73% 

91% 

67% 

n=llresponses 

11 

3 

5 

3 

2 

6 

8 

10 

7 


respondents (100%) indicated intention as one of the 
techniques used. The data collected via this prompt fur¬ 
ther identified the frequency with which faculty utilized 
intention to integrate ELOs. For example, 27% reported 
that they were intentional at least one or twice over the 
course of the semester, 46% of respondents reported that 
they were intentional between three to five times, and 
27% of respondents reported that they were intentional 
more than five times during the semester. Facilitation was 
also reported as a technique used by 91% of respondents, 
while utility was reported by 73% of respondents, aware¬ 
ness by 67%, reflection by 56% and connection by 18% of 
respondents. 

Correlations Among Data Sets 

There was a strong positive correlation between ETO in¬ 
tegration techniques used by faculty and the references 
to these themes made by faculty (n = 6) interviews (r = 
0.844, p = 0.035). See Scatterplot Representing ETO 
Integration Techniques Used Reported by Faculty and 
Corresponding References to These Themes from Faculty 
Interviews (Figure 5). 

However, there was little correlation between the learner- 
centered nature of the syllabus and increases in student- 
perceived {n = 16) ETO competence. Slight correlation 
was found between the two variables, (r = 0.350, p = 
0.184) represented by the scatterplot shown in Figure 6. 

Findings 

The results of this research study revealed that several 
themes of ETO integration support ETO competence in 
students. The use of intention and facilitation were found 
to be ETO integration themes identified and referenced 
by faculty as those that were used to support student com¬ 
petence with ETOs. In this study, students demonstrated 
growth in self-perceived competence with all six? ETOs 



Figure 5 

ELO Integration Techniques Used by 

Faculty and Faculty Interview 

References to Themes 
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Figure 6 

Scatterplot Representing 
Correlation between the 
Learner-Centered Nature of the Syllabus 
and Increases in 

Student-Perceived ELO Competence. 

o 



■ 

“a 

■ 

c 

■ 

■ 

■8 & 
a § 

■ 

co £ 

R 

■ 

■ 

£ 

■ 

a 

L. 

■ 

,R 

■ 


Syllabus Ratings Representing Learner-Centered 


Nature cf the Syllabus 


Journal of Learning in Higher Education 


89 















































Susan Cvdis. MarvLou Galantino. Carra Leah Hood. Marv Padden. & Marc Richard 


over the course of the semester. Little to no correlation 
was found between learner-centered syllabi and student 
reported ELO competence. 

DISCUSSION 

Faculty awareness of ELOs may be the first step in ELO 
integration. Integration and syllabus articulation may 
indicate intentional support for ELO competence, but it 
is necessary to go beyond this step to effectively promote 
competence in students. Since the syllabus targets mul¬ 
tiple audiences, such as faculty colleagues, program coor¬ 
dinators or chairs, Deans, faculty evaluation committees, 
accrediting agencies, in addition to students, this may be 
one of the reasons why little correlation was found be¬ 
tween content-centered syllabi and student reported gains 
in ELO competence. 

What does this mean for instructors of students in higher 
education? Simply stated, integrating essential learning 
outcomes into the teaching and learning experiences that 
make up the design of our courses should be a seamless 
process. While the content of a course is always most 
important, we can support student understanding on a 
deeper level by promoting students’ metacognitive aware¬ 
ness through making connections between what they are 
learning and the essential skills and competencies they 
need for success in the real world. 

We can begin to facilitate the connection between stu¬ 
dent learning and essential competencies by intentionally 
articulating how essential learning outcomes emerge as a 
natural outgrowth of the content we teach in any given 
course and include this information explicitly in our syl¬ 
labi. Doing so, helps to facilitate student awareness of 
their own competence as it relates to these essential skills. 
Furthermore, providing students with the opportunity to 
reflect on the confidence they develop, deepens and en¬ 
hances awareness of how they possess these essential skills. 

Although there should be a clear connection between 
course outcomes and what students are expected and 
advised to do in order to attain them, the results of this 
study show that this may not be enough to promote stu¬ 
dent learning. The instructor needs to go beyond simply 
including course outcomes in the syllabus (Habanek, 
2005). This research supports a process in which instruc¬ 
tors may need to articulate intended outcomes, making 
stated outcomes accessible and public, and use assessment 
results to improve and align instruction (Driscoll and 
Wood, 2007). While research supports the notion that 
students need to have clear expectations and understand¬ 
ing of the purpose of instruction and have progress deter¬ 
mined by achievement of learning outcomes, this study 
also suggests that communicating those outcomes via syl¬ 


labi is an important first step in focusing on ELO inte¬ 
gration, but does not alone adequately support students’ 
developing ELO competence. The syllabus alone did not 
appear to communicate to students in a substantial way. 
Although instructors might make assumptions about the 
role of syllabi in relation to pedagogy and learning, these 
assumptions might not be as certain as initially thought. 
This study supported the notion that the relational dy¬ 
namics and formative experiences of the classroom carry 
the most weight in student learning. Although syllabi 
might be quite significant when communicating to other 
audiences, it may not be as substantial for students or sup¬ 
porting the development of essential learning outcomes. 

CONCLUSIONS 

Supporting students’ competence with essential learning 
outcomes can be accomplished through a focus on ELO 
integration. As faculty engages in practices that inten¬ 
tionally integrate ELOs into the curriculum students have 
the opportunity to develop competence. The intentional 
practice of ELO integration is more likely a combination 
of factors including an intentional stance on the part of 
the instructor, and facilitation of ELO experiences. How¬ 
ever, doing so does not mean that this intentional stance 
and facilitation happens solely in the form of the design 
of the syllabus. It might actually be an interdependent 
dynamic of various features. These features include not 
only intention on the part of the instructor and facilitat¬ 
ing the learning process, but also supporting awareness on 
the part of students, creating utility of the essential skills, 
and creating opportunities for reflection as those features 
that actually promote competence. It isn’t necessarily the 
simple articulation of ELOs in the syllabus that promotes 
student competence, but rather it may be a synergetic de¬ 
pendent relationship including a focus on meaningful 
learning experiences and opportunities formative assess¬ 
ment that are key components. 

LIMITATIONS 

While the results of the analysis of syllabi showed that 
the learner-centered nature of the syllabus did not appear 
to directly support ELO competence in students, the re¬ 
searchers considered whether faculty participants provid¬ 
ed students with additional materials to support student 
awareness of ELOs such as a handout outlining additional 
explanation of the assignments to be completed in class. 
Likewise, instructors may be using a classroom server such 
as Blackboard to post additional information about learn¬ 
ing outcomes and assignment. This might be an impor¬ 
tant limitation of the results of this research. Researchers 
recognize that faculty may not have included assignment 
task descriptions with their syllabi, but they might be 
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providing this information to students via other avenues. 
These descriptions could have more meaning in relation to 
student learning than the syllabus itself. 

Programs at (the university) do not require the use of any 
specific syllabus template, therefore each instructor may 
create his or her own overview of the course. As such, this 
study did show some variation in what types of informa¬ 
tion syllabi included and communicated to students. As 
a result, there could be an important consideration with 
regard to the factors that may have impacted student ELO 
competence. 

FUTURE RESEARCH 

Opportunities for future research include collection and 
further analysis of the additional materials instructors 
may or may not be using to support ELO awareness and 
competence in students. Thus far the researchers have col¬ 
lected data from two successive semesters and have found 
similar results reported by students on the ELO self -per¬ 
ception survey. Plans for future research include collect¬ 
ing data in each academic year to further explore the re¬ 
sults longitudinally. Additionally, further research might 
include faculty using syllabi with common components 
and comparing results to the results of this study. Fur¬ 
thermore, consideration of the role of a classroom server 
for sharing information which might promote student 
competence of ELOs would be an important focus for 
further research. Additionally, researchers seek to further 
investigate direct measures of student learning to identify 
the presence of ELO competence present in artifacts col¬ 
lected as representations of tasks that promote ELO com¬ 
petence. 
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Appendix B 

Essential Learning Outcomes Survey 


Creativity and Innovation 

Level 1 

Level 2 

Level 3 

Recognize the- imp-artunits; oF 
creative pursuits-and prucesses 
within mid across disuplines. 

Capacity tu generate new ideas 
and u> uruM them tu others within 

and across various Fields. 

Independently, t>r collateral r«ely, 
synthesise ideas, knowledge, and 
experiences to devise innovative 
solutions tu complex prublems. 

Critical Thinking 

Level 1 

Level 2 

Level 3 

Knuwledge jnd CciiripriihKn&.cin 
Recall, sumin.hi/H, reris::n 
mill jin, locate and evaluate 

3D Liras. 

Application and Analysis: 
Apply,, analyte-..: ullaborate 
research arid integrate sources. 

Synthesis and ftellectiun: 
Create, translate, synthesize, 
reflect, identify and question 
assumptions. 

Program Competency 

Level 1 

Level 2 

Level 3 

Yciu will bi!::ulu !:: retugnize 
jiid/sifi-.ill theories and 

concepts. 

V:: j will be able to draw From 
previuus knowledge tu apply and 
integrate new conceptsand skills; 
connecting the old with the-new. 

You will be abletu use your 
nepuired knowledge tu evaluate 
and Djntribute tu thebodyuF 
knowledge in your field of study. 

Communication 

Level 1 

1 

Level 2 

Level 3 

You will be able tu d:s: ass 
concepts and convey ideas 
with accuracy. 

You will be able tu 

communicate ideas with 
darity, organizatiun and 
Language appropriate totlie 
audience. 

You will be able tu cummunicate 
ideas with argue izatiun and 
cuhesrceness using compelling 
language choices and pose. 

Information Literacy and Research Skill 

Level 1 

Level 2 

Level 3 

Identify, locate.. and 
evaluate informatiun on a 

basic level. 

Identify, locate, and evaluate 
iniormatiun an a bask level 
and synthesize information 
intu their own work. 

Identify, locate,. and 
evaluate information 
synthesize information into 
work and generate original 
infurmatiun. 
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Appendix B (continued) 
Essential Learning Outcomes Survey 


Teamwork and Collaboration 

Level l 

Level 2 

Level 3 

Outlier stand carrmunleahan. 
dpcrsily, consensus building, 
team £.oal setting, team protest, 
peer pncssLrEi-'onnAlcI netfanee 
anc collaboration. 

E-plam.'ccscnbc oammu nit abort. 

diversity, consensus I:l Iding. 
team peal setting, team pretest, 
peer pncssure/conflici avoidance 
and cal la bo rattan. 

3 splay/lead/adapt 
ccmmurioatlcn, diversity 
conscnsLibulcirg, team goal 
selling, learn process, peer 
pressui e/confllct abidance anc 
collaboration. 

Global Awareness 

Level 1 

Level 2 

Level 3 

GvmuiiiiliiLi JtiKiw ihJip win i 

LU a . -1 b'L ShllU■ 

jV. dl-il!fc-L-| ul L'JII'JI dl Jlhd tflllll t 

iJrti l L>i ii isra^.iciii joisJ buiulh 
Lurn—unk.il.-i iltuu lu a .ii'vi wL 
bumilikily- F'ji Ik pulu in ^urskuM 

1 ijIULUl JiVlI'inVkk. 

At-Trivl v infer m your-HIT jLjujI 
Ptlobd iuuin. Inlirrfcl jL i 
ifliir* *j:i! ukkd wnlh nJiiidirik 
jikJ/lii iirh'.r.jlki-b from -Hillin' 
lu ".u’liri. L'ifinu--il -dlh- ■.•.■] 1 111 pji v l.- 
!u Irdm Ii -lri i ulhvri win fluff bifcL 

rind ‘HhsnhmnuvL 

Hrfvu in mu ylkil jbj unihk !nl 
■In Build a ill lllUll.-Lnilh-Llvd 
hy jlum uf lu“ aluff pr<SHLHL 

LMlIIIUIlL.I jIu lllil sfCUl OW1 -MU'ld 

kivw u>d Ihu dum nunrt lJi jj in i*u_ 
iu!u'u diR xl bhflrvd lr|i il [HUfila 
Jll-d LullUIMh. 

Adapting to Change 

Level l 

Level Z 

Level 3 

Exposed to lhe pi etc-;; ol 
charge. 

Participate- in atti.itiH ihat will 
lead to adapting la change- 

Demcriirate Incepcndencc and 
critical thinking that will Imtuie 
and complete the djrgc cycle. 

Quantitative Reasoning 

Level! l 

Leve 1 2 

Level 3 

Define,, explain anc prcpcrly Lise 
fundairemal math and numeracy 
concepts and prepare- f ar mare 
advanced couisework. 

Shew a consistent ability to solve 
n-Lltl component problems., usrtg 
toih linear moihods and creative 
techmqLos. 

hluji-i imw inferuiiL on md iJaul U-, 
u". lu ri[ math u~id lunr^ fck lly: 
kjiiiiiirLDd and |uii?|u Lfixii|i hi uuio. 
ujiiI Jiiiii .4 rnJlipk hd uijkj -u jlvd 

Lh yean iilrulvdiiiBfHu ori. 

Ethi ~e 1 Reasoning 

Leve 1 1 

Level 2 

Levels 

Recogrjiean ethical cilcnma an 
understand ycLr rale in It. 

Reach a decision basee cn 
ethJcal reasoning. 

fxplari how yeu reached a 
dcclslen anc why ycL ceemed it a 
better chase ihan alterrjtlvcs. 
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Appendix C 

Faculty Post-Interview Protocol 



Welcome and thank participants for participation m the interviewing 

The interview will focus on student learning outcomes and will help us learn more about how 
to best support students with the ELO initiative. 

Your responses will be kept confidential and your name or any other information that could 
identify' you will be redacted to protect vour privacy in any reports we write as a result of this 
research. 

Concepts discovered will facilitate the research of strategies for supporting ELO integration. 


Engage participants m discussion using the questions presented below. 

Explain that the discussion will be audio recorded and notes will be recorded to document 
information collected. 

Prompt participants as appropriate using the additional prompts provided and invite 
interviewees to respond with information the}' prefer to share. 

Topic 1: 

Goals 

Ql. What are your goals for student learning in this course? 

Q2. How aware do you think students are about the goals your have for the student 
learning in this course? 

Q3. What factors do you think support student awareness of the goals of this course? 

Topic 2: 

Assessment to Guide 
Student Learning 

Q4. What assessment techniques have you used to support student learning in this 
course? 

Q5. How help fill do you think these have been in supporting student learning? 

Q6. Describe any assessment techniques that guide instruction'student learning in this 
course. 

Topic 3: 

Student Engagement 

Q7. How would you describe students' level of engagementmotivation for the 
leaming'assessment tasks in this course? 

Q8. Describe any factors that contribute to student engagement motivation for the 
learning in tbit course. 

Topic 4: 

Student Perception of 
Learning 

Q9. How effectively do you think students perceive learning in this course? 

Q10. What factors do you think contributes to student perception of learning in this 
course? 

Topic 5: 

ELO Competence 

Ql 1. How aware do you think students are about the ELOs the}' acquire from 
experiences both in and out of the classroom while at Stockton? 

Q12. Which ELOs did you address in this course? 

Q13. Which assessments in this course require students to demonstrate ELOs? 

Q14. Please describe those learning and'or assessment tasks. 

Q15. What opportunities do students have to build awareness about the ELOs the}' have 
acquired while at Stockton (from both in and out of the classroom)? 

Q16. What importance do you think student self-awareness of ELO competence has 
overall on student learning? 

Conclusion 

Invite participants to share final thoughts. 

Thank participant for interviewing. 
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Appendix D 
Faculty Survey 


ELO Study Group Faculty Survey 


1. Your Name: 

2. How often did you intentionally discuss with your students the ELOs you integrated into your course? 

Q I didn't discuss ELOs with students. 

Q 1 or 2 times over the course of the semester. 

Q 3-5 times over the course of the semester. 

Q More than 5 times over the course of the semester. 

3. Which of the following techniques did you use to support student competence with the ELOs you focused on in 
your course? {Please select all that apply.) 

I helped students connect the ELOs to other disciplines, courses or to other ELOs. 

I required/supported students with the practice of reflecting on their ELO competence. 

I helped students recognize how the ELOs are utilized in real world or practical settings. 

I identified specific learning tasks, assignments or activities to support students' ELO competence 

I focused on developing students' awareness of the ELOs and how they integrated into the learning tasks. 

Comments: 
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Appendix E 
Syllabus Rubric 


Syllabus Rubric: Scoring Sheet 

Adapted from A Syllabus Rubric (Palmer, Bach, & streifcr, 2014) 


Definitions of 'Strength of Evidence’ 

• Strong evidence indicates that many {but not necessarily all) of the characteristics of the 
component are present in the syllabus and match the descriptions closely. 

• Moderate evidence indicates that only some of the characteristics of the component are 
present in the syllabus and/or only partly match the descriptions. 

• Low evidence indicates that very few or none of the characteristics of the component are 
present in the syllabus and/or don't match the descriptions. 



Component 


ssenbal learning Outcomes are dcariy Identified 


a. Objectives and assessments are algned 



cssmcnts are adequately paced ard sraltaldct: 


8. Tool tor assessment Is aligned with ELOs, objectives 
and assessments 


9. Classroom artlotlcs, assessments, and cb.ccbvcs 
arc algned 


10. Learning activities arederned from evidence based 
practices 


11. Learning activities likely to acbvcty engage students 


12. Positive, respectful in, I trig. Includes the use of 
prenouns such a 'you' to convey posltne tone. 


13. Fosters motlvallcn tor meaningful engagement 
value of course In post course life, promotes 
content as a vehicle for learning 


1A. Communicates high cupectailons, projects 
confidence of success 


IS. S^labus is well organized, easy to navigate, requires 



subtotals |x3| ■ 


1*21 ■ 

TOTAL 


content-centered 


transitional 


Palmer, M. S., Bach. D. J.. & A_ C. (2014). Measuring the preanise: A learning-focused syllabus rubric. To 

improve tile academy: A journal of educaboaal dcvefopmeal, 33 (1 j, 14-36. 
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